<html>
<head><meta charset="utf-8"><title>atomic load vs rmw · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html">atomic load vs rmw</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="208005845"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208005845" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208005845">(Aug 25 2020 at 18:50)</a>:</h4>
<p>I've been playing with loom lately, and I found it surprising that it seems like<br>
RMW and load operations have different semantics, even for seemingly trivial<br>
programs:</p>
<div class="codehilite"><pre><span></span><code><span class="cp">#[test]</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">test_load</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">loom</span>::<span class="n">model</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">v1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">AtomicUsize</span>::<span class="n">new</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">v2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>

<span class="w">        </span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="n">dbg</span><span class="o">!</span><span class="p">(</span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">));</span><span class="w"></span>
<span class="w">        </span><span class="p">});</span><span class="w"></span>

<span class="w">        </span><span class="n">dbg</span><span class="o">!</span><span class="p">(</span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="p">});</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="cp">#[test]</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">test_rmw</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">loom</span>::<span class="n">model</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">v1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">AtomicUsize</span>::<span class="n">new</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">v2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>

<span class="w">        </span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="n">dbg</span><span class="o">!</span><span class="p">(</span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">));</span><span class="w"></span>
<span class="w">        </span><span class="p">});</span><span class="w"></span>

<span class="w">        </span><span class="n">dbg</span><span class="o">!</span><span class="p">(</span><span class="n">v2</span><span class="p">.</span><span class="n">fetch_or</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="p">});</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>


<p>These two tests produce different results; the load operation has 4 possible<br>
outcomes according to loom, while the rmw test has just 3. In particular, the<br>
load has the following possible path:</p>
<div class="codehilite"><pre><span></span><code>~~~~~~~~ THREAD 1 ~~~~~~~~
v1.store(1, SeqCst) = ()
~~~~~~~~ THREAD 0 ~~~~~~~~
v2.load(SeqCst) = 0
</code></pre></div>


<p>whereas the rmw write operation has no such case; it always sees the 1 stored by<br>
thread 1. Note that I definitely <em>expect</em> that a single load may not see the 1<br>
stored in the other thread. But I find it unexpected that the rmw operation does<br>
not encounter this case at all.</p>
<p>I am trying to understand whether this is a limitation of loom or if there's<br>
something in Rust's memory model (or, I guess, C11 or whatever) that I could<br>
read to better understand this case.</p>
<p>Maybe to clarify a bit more, before finding this, I think I more or less<br>
expected that a program like the following would eventually terminate (modulo<br>
just never scheduling the first thread or something like that). But, AFAICT, at<br>
least a cursory reading of the documentation I was able to find seems to suggest<br>
this is not the case, i.e., there is no guarantee that the load will ever see a<br>
store issued on another thread. Is that true? Is there some way to say "no<br>
please synchronize with the other thread"?</p>
<div class="codehilite"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">v1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">AtomicUsize</span>::<span class="n">new</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">v2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>

<span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">);</span><span class="w"></span>
<span class="p">});</span><span class="w"></span>

<span class="k">while</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
</code></pre></div>



<a name="208006192"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208006192" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208006192">(Aug 25 2020 at 18:53)</a>:</h4>
<p>I would also expect that <code>test_rmw</code> could see 0</p>



<a name="208008016"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208008016" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208008016">(Aug 25 2020 at 19:04)</a>:</h4>
<p>There is this snippet in loom which is explicitly noting the difference between rmw and load operations (<a href="https://github.com/tokio-rs/loom/blob/master/src/rt/atomic.rs#L724-L725">https://github.com/tokio-rs/loom/blob/master/src/rt/atomic.rs#L724-L725</a>) but it was introduced in a huge refactor and I don't see any commentary beyond that.</p>



<a name="208023871"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208023871" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208023871">(Aug 25 2020 at 21:09)</a>:</h4>
<p>Looking more carefully at the results:</p>
<div class="codehilite"><pre><span></span><code><span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">fetch_or</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>

<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
<span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">fetch_or</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>

<span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">fetch_or</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
</code></pre></div>


<div class="codehilite"><pre><span></span><code><span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>

<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
<span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w">  </span><span class="c1">// &lt;-- bad</span>

<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
<span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>

<span class="cm">/* T2 */</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="cm">/* T1 */</span><span class="w"> </span><span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
</code></pre></div>


<p>I don't know much about how loom works but I would guess that the order of printing matches the seqcst order, in which case it is the rmw that is correct and not the load: a load should not read 0 after 1 has been written earlier in seqcst order</p>



<a name="208023991"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208023991" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208023991">(Aug 25 2020 at 21:10)</a>:</h4>
<p>the result where the RMW reads 0 and is executed first is already represented (twice) in the other two cases</p>



<a name="208025062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208025062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208025062">(Aug 25 2020 at 21:18)</a>:</h4>
<p>It is unclear whether the (as yet not fully defined) weak memory model supports the progress guarantees necessary to claim that a basic message passing spinlock will eventually terminate. As far as I can tell, C11 seems to both promise progress in the abstract but also not provide any concurrent progress guarantees. I don't think that sequential consistency implies starvation freedom</p>



<a name="208025325"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208025325" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208025325">(Aug 25 2020 at 21:21)</a>:</h4>
<p>However the code you wrote will certainly terminate on all real architectures (unless they choose not to run the thread at all; that might be up to the OS to some extent)</p>



<a name="208026075"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208026075" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208026075">(Aug 25 2020 at 21:27)</a>:</h4>
<p>In fact, I think they might want to not guarantee this, since such a promise would invalidate sequentialization - the code</p>
<div class="codehilite"><pre><span></span><code><span class="k">while</span><span class="w"> </span><span class="n">v2</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">SeqCst</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
<span class="n">v1</span><span class="p">.</span><span class="n">store</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">SeqCst</span><span class="p">);</span><span class="w"></span>
</code></pre></div>


<p>on one thread will diverge while the original program must not, assuming an eventual progress guarantee in the semantics. In general I think a lot of compiler transformations can possibly add synchronizations leading to deadlock</p>



<a name="208031426"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031426" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031426">(Aug 25 2020 at 22:19)</a>:</h4>
<p><span class="user-mention" data-user-id="271719">@Mario Carneiro</span> do you know if there is a standard-guaranteed way to implement this sort of message passing type of atomics to work, not just "in practice"? It seems like a fundamental abstraction that should have some way of implementing it.</p>
<p>To be clear, ideally, this wouldn't necessarily be a spinlock in the parent/main thread, it might be something like a select() call that checks an atomic variable after receiving a readiness notification (e.g., a vector length). My current understanding says that C11 does not guarantee that such a program will ever observe that the other thread incremented the length of said vector. This sort of lack of "concurrent progress" seems painful to write correct code against, if you want to avoid hand-waving and saying "But in reality progress <em>is</em> guaranteed".</p>



<a name="208031518"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031518" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031518">(Aug 25 2020 at 22:20)</a>:</h4>
<p>What you have written is definitely the "approved" way to do it, if anything works then that should</p>



<a name="208031536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031536">(Aug 25 2020 at 22:20)</a>:</h4>
<p>I'm just not sure if there are <em>any</em> liveness guarantees along these lines in the standard</p>



<a name="208031566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031566">(Aug 25 2020 at 22:21)</a>:</h4>
<p>Is that a known limitation? I feel like I can't be the first one to stumble across this, but have not seen anything to that effect anywhere.</p>



<a name="208031584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031584">(Aug 25 2020 at 22:21)</a>:</h4>
<p>There are a couple things under the name "progress" in the C++ standard but I'm not sure how much of that applies in a concurrent setting</p>



<a name="208031616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031616">(Aug 25 2020 at 22:21)</a>:</h4>
<p>I know that this has historically been not the focus in research on weak memory models</p>



<a name="208031686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031686">(Aug 25 2020 at 22:22)</a>:</h4>
<p>which are generally more concerned with safety than liveness</p>



<a name="208031724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031724">(Aug 25 2020 at 22:22)</a>:</h4>
<p>That seems... surprising to me, in some sense.</p>
<p>Are you aware of good places to ask this sort of question? I guess I should file a bug at loom at least</p>



<a name="208031765"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208031765" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208031765">(Aug 25 2020 at 22:23)</a>:</h4>
<p><span class="user-mention" data-user-id="120791">@RalfJ</span> knows more about this research than me, maybe he can help</p>



<a name="208032451"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208032451" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208032451">(Aug 25 2020 at 22:31)</a>:</h4>
<p>Aha, I think this will do the job - <a href="https://timsong-cpp.github.io/cppwp/n4659/intro.progress#18">https://timsong-cpp.github.io/cppwp/n4659/intro.progress#18</a></p>



<a name="208032542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208032542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208032542">(Aug 25 2020 at 22:32)</a>:</h4>
<p>Aha! I guess that's ... C++17?</p>



<a name="208032558"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208032558" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208032558">(Aug 25 2020 at 22:33)</a>:</h4>
<p>Not that it matters much, I suppose.</p>



<a name="208032699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208032699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208032699">(Aug 25 2020 at 22:35)</a>:</h4>
<p>C++11 too, it is 1.10.25 at <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3242.pdf">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3242.pdf</a></p>



<a name="208033105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208033105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208033105">(Aug 25 2020 at 22:39)</a>:</h4>
<p>This does seem helpful, thank you.</p>



<a name="208033168"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208033168" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208033168">(Aug 25 2020 at 22:40)</a>:</h4>
<p><span class="user-mention" data-user-id="271719">@Mario Carneiro</span> you agree that I should likely file a bug with upstream loom, right?</p>



<a name="208033324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208033324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208033324">(Aug 25 2020 at 22:41)</a>:</h4>
<p>I think so. I don't know where the issue is but it's clearly not internally consistent</p>



<a name="208033610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208033610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208033610">(Aug 25 2020 at 22:44)</a>:</h4>
<p><a href="https://github.com/tokio-rs/loom/blob/master/src/rt/atomic.rs#L705">This line</a> seems to say that a seqcst load should only load the latest seqcst write if that is all that is available</p>



<a name="208060090"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060090" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060090">(Aug 26 2020 at 07:26)</a>:</h4>
<p>RMWs have much stronger ordering guarantees, so this does not sound surprising to me. I haven't looked into all the details though.</p>



<a name="208060147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060147">(Aug 26 2020 at 07:27)</a>:</h4>
<p>oh, these are all SeqCst accesses? sorry I thought it would be release-acquire as that makes most sense most of the time^^</p>



<a name="208060449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060449">(Aug 26 2020 at 07:31)</a>:</h4>
<blockquote>
<p>whereas the rmw write operation has no such case; it always sees the 1 stored by<br>
thread 1. Note that I definitely expect that a single load may not see the 1<br>
stored in the other thread. But I find it unexpected that the rmw operation does<br>
not encounter this case at all.</p>
</blockquote>
<p>not sure what that path means... the usual way to ask about possible behaviors in a weak memory program is to look at all the reads and see which values they can see. You seem to also look at the "order" here which is meaningless in general. Also you are doing printing and I have no idea how that might affect ordering, certainly you should not rely on the printing in the <code>spawned</code> thread happening atomically at the same time as the actual store... some time later it'll print, but that means nothing.</p>
<p>So let's ignore the <code>dbg!</code> in the spawned thread. But I agree that the other <code>dbg!</code> can print either 0 or 1 in both programs. This is uncorrelated with the order of the two prints for the reasons laid out above.</p>



<a name="208060592"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060592" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060592">(Aug 26 2020 at 07:33)</a>:</h4>
<p>well loom is a model, so it's probably controlling the order in which atomic operations succeed. So the printing order could well be relevant</p>



<a name="208060627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060627">(Aug 26 2020 at 07:33)</a>:</h4>
<p>my presumption is that it represents the SeqCst order</p>



<a name="208060679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060679">(Aug 26 2020 at 07:34)</a>:</h4>
<blockquote>
<p>Maybe to clarify a bit more, before finding this, I think I more or less<br>
expected that a program like the following would eventually terminate (modulo<br>
just never scheduling the first thread or something like that).</p>
</blockquote>
<p>Oh my... now you are talking about liveness. In a concurrency setting. The C++ standard indeed says some things there as <span class="user-mention" data-user-id="271719">@Mario Carneiro</span> mentioned, though I don't know what it says. AFAIK there is hardly any research on actually underpinning that wording with anything formal. Researchers are still busy figuring out safety in a weak memory setting; liveness is years off I think.^^</p>
<p>But it is certainly the <em>intention</em> of the C++ model that such a program will eventually terminate, and in practice this is something you should be able to rely on.</p>



<a name="208060731"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060731" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060731">(Aug 26 2020 at 07:34)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/atomic.20load.20vs.20rmw/near/208060592">said</a>:</p>
<blockquote>
<p>well loom is a model, so it's probably controlling the order in which atomic operations succeed. So the printing order could well be relevant</p>
</blockquote>
<p>I dont know how loom works, but I dont see why it would make the printing atomic with the store. It might as well execute the store in the spanwed thread, then go on with the main thread, and at the end do the printing in the spawned thread.</p>



<a name="208060759"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060759" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060759">(Aug 26 2020 at 07:35)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/atomic.20load.20vs.20rmw/near/208060627">said</a>:</p>
<blockquote>
<p>my presumption is that it represents the SeqCst order</p>
</blockquote>
<p>loom supports weak memory, where most operations are not part of the SeqCst order. so this cannot be the case in general.</p>



<a name="208060872"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060872" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060872">(Aug 26 2020 at 07:36)</a>:</h4>
<p>FWIW the printing order does not change when I run the test many times</p>



<a name="208060909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060909">(Aug 26 2020 at 07:37)</a>:</h4>
<p>which shows nothing as this is non-determinism, you might as well get the same execution each time</p>



<a name="208060929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060929">(Aug 26 2020 at 07:37)</a>:</h4>
<p>right...</p>



<a name="208060992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208060992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208060992">(Aug 26 2020 at 07:38)</a>:</h4>
<p>it is impossible to figure out the order of the store and the load directly (except maybe with a debugger, but that will completely wreck havok with any possible weak behaviors). the only way to observe their order is indirectly, by checking if the load got a 0 or a 1.</p>



<a name="208061021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061021">(Aug 26 2020 at 07:38)</a>:</h4>
<p>that's why in every single weak memory paper, "litmus tests" are written as programs with a bunch of stores, RMW, and loads, and then specified by "what are the possible sets of values that those loads might see"</p>



<a name="208061107"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061107" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061107">(Aug 26 2020 at 07:40)</a>:</h4>
<p>Still, it is an exhaustive enumeration of all possible behaviors. I assume it's doing a prolog like backtracking thing; it would be pretty weird if it were nondeterministic</p>



<a name="208061131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061131">(Aug 26 2020 at 07:40)</a>:</h4>
<p>that is the only meaningful way to talk about the possible behaviors of weak memory programs. no total order exists for all operations, and for the special few operations that are subject to a total order (SeqCst, and also the order of writes <em>on a single location</em>), the order is not observable except by checking what values a load can see.</p>



<a name="208061181"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061181" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061181">(Aug 26 2020 at 07:41)</a>:</h4>
<p>does loom really enumerate all possible interleavings of printing? AFAIK it is focused on producing all possible values for all loads, and that's it.</p>



<a name="208061202"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061202" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061202">(Aug 26 2020 at 07:41)</a>:</h4>
<p>printing itself forces another total order onto the program which heavily interferes with whatever the memory model does</p>



<a name="208061281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061281">(Aug 26 2020 at 07:42)</a>:</h4>
<p>so trying to measure things by printing in the middle of a weak memory execution is like trying to measure wildlife by putting spotlights into the forest. sure, you might see something, but it has nothing to do with the thing you want to actually measure.</p>



<a name="208061322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208061322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208061322">(Aug 26 2020 at 07:43)</a>:</h4>
<p>to measure memory models, you have to first collect all the data (do all the loads and stores you are interested in), and only at the end start printing them out. that is the only way to be sure that the printing does not interfere with your measurement.</p>



<a name="208080541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208080541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208080541">(Aug 26 2020 at 11:43)</a>:</h4>
<p>loom single-threads the program so printing should be fine I think</p>



<a name="208184826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208184826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208184826">(Aug 27 2020 at 07:19)</a>:</h4>
<p>eh... but it has to explore all interleavings, right?</p>



<a name="208184850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208184850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208184850">(Aug 27 2020 at 07:19)</a>:</h4>
<p>like, Miri also "single-threads" the program. but that's in the same sense that all programs are single-threaded when you only have 1 physical core.</p>



<a name="208184903"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208184903" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208184903">(Aug 27 2020 at 07:20)</a>:</h4>
<p>it changes nothing about what I said above. the print is still not atomic with the store.</p>



<a name="208184915"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/atomic%20load%20vs%20rmw/near/208184915" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/atomic.20load.20vs.20rmw.html#208184915">(Aug 27 2020 at 07:20)</a>:</h4>
<p>also no idea if loom models the synchronization implicit in printing; if it does that would also still be a problem</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>